Discrete and continuous representations and processing in deep learning: Looking forward

Discrete and continuous representations of content (e.g., language or images) have interesting properties to be explored for the understanding reasoning with this by machines. This position paper puts forward our opinion on role discrete their processing in deep learning field. Current neural network models compute continuous-valued data. Information is compressed into dense, distributed embeddings. By stark contrast, humans use symbols communication language. Such represent a version world that derives its meaning from shared contextual information. Additionally, human involves symbol manipulation at cognitive level, which facilitates abstract reasoning, composition knowledge understanding, generalization efficient learning. Motivated these insights, we argue combining will essential build systems exhibit general form intelligence. We suggest discuss several avenues could improve current networks inclusion elements combine advantages both types representations.